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Abstract: The main research object of the paper is investigation and proposal of the comprehensive Learning 
Object Repositories (LORs) quality evaluation tool suitable for their multiple criteria decision analysis, evaluation 
and optimisation. Both LORs ‘internal quality’ and ‘quality in use’ evaluation (decision making) criteria are 
analysed in the paper. The authors have analysed several well-known LORs quality evaluation tools. In their 
opinion, the comprehensive multiple criteria LOR quality evaluation tool should include both general software 
‘internal quality’ evaluation criteria and ‘quality in use’ evaluation criteria suitable for the particular project or user. 
In the authors’ opinion, the proposed LOR ‘Architecture’ group criteria are general ‘internal quality’ evaluation 
criteria, and ‘Metadata’, ‘Storage’, ‘Graphical user interface’ and ‘Other’ are ‘customisable’ ‘quality in use’ 
evaluation criteria. The authors have also presented their comprehensive Virtual Learning Environments (VLEs) 
quality evaluation tool combining both ‘internal quality’ (i.e., ‘General Architecture’) and ‘quality in use’ (i.e., 
‘Adaptation’) technological evaluation criteria. The authors have proposed to use the quality evaluation rating tool 
while evaluating LORs and VLEs. The authors have analysed that if we want to optimise LORs and VLEs (or the 
other learning software packages) for the individual learner needs, i.e., to personalise his/her learning process in 
the best way according to their prerequisites, preferred learning speed and methods etc., we should use the 
experts’ additive utility function including the proposed LORs and VLEs expert evaluation criteria ratings together 
with the experts preferred weights of evaluation criteria. In this case we have the multiple criteria optimisation 
task using criteria ratings, and their weights. Quality evaluation criteria of the main e-Learning system 
components, i.e., LORs and VLEs are further investigated as the possible learning software packages 
optimisation parameters. Scalarization method is explored in the paper to be applied to optimise the learning 
software packages according to the individualised learners needs. Several open source VLEs evaluation results 
are also presented in the paper. 

Keywords: managing quality in e-learning, multiple criteria evaluation, learning object repositories, virtual 
learning environments, optimisation 

1. Introduction: Problem of multiple criteria evaluation of learning software 
packages 

1 .1 Problem of quality evaluation of learning object repositories 

One of the largest LORs related R&D projects at the moment is EdReNe. EdReNe - Educational 
Repositories Network - brings together web-based repositories of learning objects with content 
owners and other stakeholders within education in order to share, develop and document strategies, 
experiences, practices, solutions, advice, procedures etc. on the organisation, structuring and 
functionality of repositories (EdReNe 2009). The network invites new associate members. The second 
project implementation year has started by the strategic seminar in Lisbon on June 9- 11, 2008. The 
main aim of this seminar was to collaboratively establish priorities for the following project 
implementation period covering all the main project themes: 

■ Repositories strategies and resources. 

■ Standards and interoperability. 

■ Engagement of producers and users - features and functionality. 

■ Rights issues. 

The LORs quality assurance strategies priority was ranked the highest by the EdReNe and external 
experts in Lisbon: 58.8 % experts have ranked it as essential, and 38.3 % - as important (Kurilovas 
and Kubilinskiene 2008). 
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The main problem here is investigation and proposal of the LORs quality evaluation model (tool) and 
method suitable for the development of LORs quality assurance strategies. The paper aims to 
analyse this LORs quality evaluation problem in more detail. 

1.2 Methods of multiple criteria evaluation of learning software packages 

In ISO/IEC 14598-1:1999 quality evaluation is defined as “the systematic examination of the extent to 
which an entity (part, product, service or organisation) is capable of meeting specified requirements”. 

Learning object (LO) is referred in the paper as “any digital resource that can be reused to support 
learning” (Wiley 2000). 

LO repositories (LORs) are considered here as properly constituted systems (i.e., organised LOs 
collections) consisting of LOs, their metadata and tools/services to manage them (Kurilovas 2007). 

Metadata is considered here as “structured data about data” (Duval et. al. 2002). 

Researchers usually divide quality evaluation criteria of e-Learning systems components into the 
technological, pedagogical and organisational evaluation criteria (Kurilovas 2007; Kurilovas and 
Dagiene 2008). Technological quality criteria are relevant for all e-Learning systems components, 
while pedagogical, organisational and intellectual property rights (licensing schemas etc.) criteria are 
mostly relevant for LOs and VLEs. 

We can evaluate both ‘internal quality’ and ‘quality in use’ of the components of e-Learning systems 
such as LORs and VLEs. According to Gasperovic and Caplinskas (2006), ‘internal quality’ is a 
descriptive characteristic that describes the quality of software independently from any particular 
context of its use. ‘Quality in use’ is evaluative characteristic of software obtained by making a 
judgment based on criteria that determine the worthiness of software for a particular project. 
Gasperovic and Caplinskas (2006) consider that it is impossible to evaluate ‘quality in use’ without 
knowing characteristics of ‘internal quality’. 

The scientists who have explored the quality of software consider that there exists no simple way to 
evaluate the functionality characteristics of ‘internal quality’ of software. According to Gasperovic and 
Caplinskas (2006), it is a hard and complicated task, which requires relatively high time and labour 
overheads. 

There are a number of scientific methods suitable for evaluation of the learning software packages. 
The paper is aimed to consider the problems of expert evaluation of the technological quality criteria 
of e-Learning system components. 

Expert evaluation is referred to here as the multiple criteria evaluation of e-Learning system 
components aimed at the selection of the best alternative based on score-ranking results. According 
to Dzemyda and Saltenis (1994), if the set of decision alternatives is assumed to be predefined, fixed 
and finite, then the decision problem is to choose the optimal alternative or, maybe, to rank them. But 
usually the experts (decision makers) have to deal with the problem of optimal decision in the multiple 
criteria situation where the objectives are often conflicting. In this case, according to Dzemyda and 
Saltenis (1994), an optimal decision is the one that maximises the decision maker’s utility. According 
to Zavadskas and Turskis (2008), each alternative in multi-criteria decision making problem can be 
described by a set of criteria. Criteria can be qualitative and quantitative. They usually have different 
units of measurement and different optimisation direction. 

We can use several vector optimization methods for multiple criteria evaluation of quality and 
optimization of learning software packages (i.e., LORs and VLEs). Since solutions of vector 
optimization problems can be characterized and computed as solutions of appropriate scalar 
optimization problems, application of scalarization method using the experts’ additive utility function is 
investigated in more detail. 

The rest of the paper is organised as follows. Sections 2 presents the several well-known LOR quality 
evaluation tools. Section 3 summarises the literature analysis results. Section 4 presents the authors’ 
comprehensive LOR quality evaluation tool based on the analysis of Sections 2 and 3, Section 5 
presents VLEs quality parameters and the several popular open source VLEs quality evaluation 
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results, Section 6 presents the LORs and VLEs quality optimisation methods. Conclusion and results 
are provided in Section 7. 

2. Quality evaluation criteria of learning object repositories: literature analysis 

2.1 SWITCH evaluation grid of the quality of learning object repository 

The first LORs quality evaluation tool (set of criteria) analysed here is the SWITCH project tool 
(SWITCH 2008) presented while evaluating DSpace and Fedora LORs in 2008. The SWITCH LOR 
evaluation tool has examined the five LOR quality criteria groups: 

- Architecture: Flexible and modular system; Possibility to use LOR system as part of a federation; 
API for storage engine; API for user access rights; API for federation functions; Metadata search 
with heterogeneous schemas; Full-text search; Performance; Scalability; Security; Interoperability; 
Persistent links; Internationalisation. 

■ Metadata: Minimal metadata schema; Predefined sets of metadata; Customisable metadata 
schema; Metadata mapping for metadata search; Unicode support; Social tagging. 

■ Graphical user interface: Complete standard Ul; Extensible standard Ul; Multiple standard Uls; 
Custom Uls; AAI authentication; Associate copyright license; Direct distribution; Direct streaming; 
Alternative protocols for data upload. 

■ Storage: Object can be of any format; Multi-part objects; Access rights; Hierarchical organisation; 
Property and metadata inheritance; Versioning system; Large objects. 

■ Other: Strength of development community; Strength of users’ community; Code quality; 
Documentation quality; Ease of installation. 

2.2 CatalystIT criteria for technical evaluation of open source repositories 

The second LOR quality evaluation tool analysed here is the tool presented by CatalystIT while 
evaluating DSpace, EPrints and Fedora LORs in (Technical... 2006). 

Two importance ratings were used by the evaluators here: self configuring (DIY) solution and Hosted / 
Hub solution rated from 0 to 4. 

This tool has examined ten LOR quality criteria: 

■ Scalability: Scale Up; Scale out; Architecture. 

■ Ease of working on code base: Add / change digital object type; Documentation of code and code 
consistency & style. 

■ Security: Data encryption; Server security; Authentication; Authorisation / access rights; Ability to 
restrict access at repository item level. 

■ Interoperability: OAI-PMH compliance; SOAP, UDDI; SRU / SRW; Bulk import and export; 
Institution exit mechanism to withdraw their content from the repository farm; Authentication; 
Standard metadata. 

- Ease of deployment: Software and hardware requirements; Packaging and installation steps; 
Separate repository and branding for each institution. 

■ System administration: Ability to customise look and feel; Ease of publishing. 

■ Internationalisation: Localisable Ul; Unicode text editing and storage. 

■ Open source: Open source license; Defined roadmap for the future. 

- Workflow tools: Workflow integration; Support for different workflows. 

■ Community knowledge base: Quality and completeness of information on the product's web site; 
Size of and level of activity in the developer community; Size of and level of activity in the user 
community; Availability and use of a range of communication channels; Software release history 
for evidence of sustainability and vitality; Documentation on how to set up and manage a 
repository farm. 

The quality evaluation ratings / values used in this tool are based on the analysis of the level of the 
criterion / feature support (see Table 1). This level of support could be also estimated as the level of 
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modification measured, e.g., by time and money needed to reach the desired level of support of the 
criterion or feature. 


Table 1: Quality evaluation ratings / values (Kurilovas and Dagiene 2008) 


Rating Explanation 


0 

Failed or feature does not exist 

1 

Has poor support and / or it can be done but with significant effort 

2 

Fair support but needs modification to reach the desired level of support 

3 

Good support and needs a minimal amount of effort 

4 

Excellent support and meets the criteria out of the box, minimal effort 


Each selected criterion is proposed to be given an importance rating to be used when evaluating the 
e-Learning system components (e.g., LOs, LORs and VLEs). Major criteria have to be broken down 
into sub-criteria with each sub-criterion also having an importance rating. The importance rating range 
is 0-4, with 0 being the lowest and 4 being of the highest importance. Each sub-criterion has then to 
be rated using a range of 0-4. 

2.3 The OMIl software repository evaluation criteria 

The next tool analysed here is “Software Repository - Evaluation Criteria and Dissemination” 
(Newhouse 2005). This OMIl document has specified the three critical phases of the LOR process: 

■ The information that must be captured when a product is created within the repository and a 
specific release submitted to the repository. 

■ The assessment criteria that should be used to review the software contribution. 

■ How the product and release information, coupled with the evaluation results, are presented 
within the repository. 

The tool combines the three types of criteria: documentation, technical and management. 

■ Documentation: Introductory docs; Pre-requisite docs; Installation docs; User docs; Admin docs; 
Tutorials; Functional specification; Implementation specification; Test documents. 

■ Technical: Pre-requisites; Deployment; Verification; Stability; Scalability; Coding. 

■ Management: Support; Sustainability; Standards. 

The evaluation points in this tool cover several areas. The following key applies to the table: 

- Criteria (the aspect under consideration). 

- Description (a description of the criteria). 

■ Form: How the information is captured (Score: A 0-5 radio button. 0: Not applicable. 1-5 numerical 
assessment. A high score is better (i.e., nearer to the ideal)). 

■ Ideal (what must be achieved to get a score of 5). 

3. Conclusions of the literature analysis and problems to solve 

The authors have analysed more LOR quality evaluation criteria based on the different LO life cycle 
stages in (Kurilovas and Dagiene 2008). There is a number of LO quality evaluation criteria before LO 
inclusion in LOR but they are out of the scope of this paper. But there are also a number of quality 
evaluation criteria during and after LO inclusion in the LOR, and these criteria reflect the quality of the 
LOR itself. They are: 

■ Criteria during LO inclusion in the LOR (Membership of contribution control strategies: using LO 
harvesting, obligatory membership; Technical interoperability: automatic verification of capability 
with known protocols; automatic metadata generation or simplified metadata tagging). 

■ Criteria after LO inclusion in the LOR (Retrieval quality: user should be able to retrieve LO in 
different ways; Information quality: display information strategies, feedback techniques). 

In the Introductory Section the authors have analysed that there exist both ‘internal quality’ and 
‘quality in use’ evaluation criteria of the components of e-Learning systems (such as LORs). 

In the authors’ opinion, no one of the tools presented in Section 2 has clearly divided the LORs quality 
evaluation criteria into two separate groups: LORs ‘internal quality’ evaluation criteria and ‘quality in 
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use’ evaluation criteria. Therefore it is hard to understand which criteria reflect the basic LORs quality 
aspects suitable for all LOR software packages alternatives, and which are suitable only for the 
particular project or user, and therefore need the users’ feedback. 

4. The comprehensive quality evaluation tool of learning object repository 

While analysing the LOR quality evaluation criteria presented in Sections 2 - 3 we can notice that the 
several tools pay more attention to the general software ‘internal quality’ evaluation criteria (such as 
the ‘Architecture’ group criteria) and the several - to the ‘customisable’ ‘quality in use’ evaluation 
criteria groups suitable for the particular project or user: ‘Metadata’, ‘Storage’, ‘Graphical user 
interface’ and ‘Other’. 

In the authors’ opinion, the comprehensive LOR quality evaluation tool should include both the 
general software ‘internal quality’ evaluation criteria suitable for all the software packages evaluation 
cases, and the ‘quality in use’ evaluation criteria suitable for the particular project or user. The authors 
propose to separately analyse the ‘internal quality’ criteria incorporated into the ‘Architecture’ criteria 
group, and the ‘quality in use’ criteria incorporated into the four groups: ‘Metadata’, ‘Storage’, 
‘Graphical user interface’ and ‘Other’. 


The authors’ comprehensive LOR quality evaluation tool is presented in Figure 1. 

Flexibility and modularity of the LOR system 



This tool is mostly similar to the SWITCH tool (see Section 2.1) in comparison with the other tools 
presented in Sections 2.2 and 2.3, but it also includes criteria from the other presented tools (incl. the 
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authors’ own research). The main ideas for the constitution of this model are to clearly divide LORs 
quality evaluation criteria according to Gasperovic and Caplinskas (2006), as well as to ensure the 
comprehensiveness of the model and to avoid the overlap of the criteria. 

The overlapping criteria here could be ‘Accessibility: access for all’ (it could also be included into the 
‘Architecture’ group, but this criterion also needs users’ evaluation, therefore it is included into ‘Quality 
in use’ criteria group), ‘Full text search’ (which could also be included into the ‘Quality in use’ criteria 
group), and ‘Property and metadata inheritance’ (it could also be included into the ‘Metadata’ group, 
but it deals also with the ‘Storage’ issues). We have mentioned 34 different evaluation criteria in this 
model (set of criteria), from which 11 criteria deal with ‘Internal quality’ (or ‘Architecture’), and 23 
criteria deal with ‘Quality in use’. 23 ‘Quality in use’ criteria are divided into four groups for the 
probably higher quality of practical evaluation and convenience reasons. There could be different 
experts (engineers, programmers and users) for different groups of ‘Quality in use’ criteria: ‘Metadata’, 
‘Storage’ and ‘Graphical user interface’ criteria need different kind of the evaluators’ expertise. 

We expect that the advantages of the proposed model could be its comprehensiveness and the clear 
division of the criteria (Kurilovas 2009). Therefore, this model could provide the experts of e-learning 
sector the clear instrumentality who (i.e., what kind of experts) should analyse what kind of LQRs 
quality criteria in order to select the best LQR software package suitable for their needs. Software 
engineering experts should analyse ‘internal quality’ criteria based on scientific informatics 
engineering knowledge, and programmers and users should analyse ‘quality in use’ criteria based on 
the users’ feedback. 

5. Evaluation of quality of virtual learning environments 

The authors have presented their comprehensive VLE technological quality evaluation tool in 
(Kurilovas and Dagiene 2008). It is based mainly on two well-known VLE evaluation methods. These 
analysed methods are: 

■ Methodology of technical evaluation of learning management systems (LMSs) (Technical... 
2004). 

■ Method of evaluation of open source e-learning platforms with the main focus on adaptation 
issues (Graf and List 2005). 

5.1 Methodology of technical evaluation of learning management systems 

Methodology of Technical Evaluation of LMSs (or VLEs) is a part of the Evaluation of Learning 
Management Software activity undertaken as part of the New Zealand Qpen Source VLE project 
(Technical... 2004). 

The evaluation criteria expand on a subset of the criteria, focusing on the technical aspects of VLEs: 

■ Qverall architecture and implementation (‘suitable for technological evaluation - authors’ 
comment): Scalability of the system; System modularity and extensibility; Possibility of multiple 
installations on a single platform; Reasonable performance optimisations; Look and feel is 
configurable; Security; Modular authentication; Robustness and stability; Installation, 
dependencies and portability. 

■ Interoperability (‘suitable for technological evaluation - authors’ comment): Integration is 
straightforward; LMSA/LE standards support. 

■ Cost of ownership. 

■ Strength of the development community: Installed base and longevity; Documentation; End-user 
community; Developer community; Qpen development process; Commercial support community. 

■ Licensing. 

■ Internationalisation and localisation (‘suitable for technological evaluation - authors’ comment): 
Localisable user interface; Localisation to relevant languages; Unicode text editing and storage; 
Time zones and date localisation; Alternative language support. 

- Accessibility (‘suitable for technological evaluation - authors’ comment): Text-only navigation 
support; Scalable fonts and graphics. 

■ Document transformation (Technical... 2004; Kurilovas 2005). 
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5.2 Virtual learning environments adaptation evaluation instrument 

Graf and List (2005) paper presents an evaluation of open source e-learning platforms / VLEs with the 
main focus on adaptation issues - adaptability, personalisation, extensibility, and adaptivity 
capabilities of the platforms. Graf and List (2005) research is focused on customisable adaptation 
only, which can be done without programming skills. These VLEs adaptation criteria are: 

■ Adaptability - includes all facilities to customise the platform for the educational institution’s needs 
(e.g., the language or the design). 

■ Personalisation aspects - indicate the facilities of each individual user to customise his/her own 
view of the platform. 

■ Extensibility - is, in principle, possible for all open source products. Nevertheless, there can be 
big differences. For example, a good programming style or the availability of a documented 
application programming interfaces (API) are helpful. 

■ Adaptivity - indicates all kinds of automatic adaptation to the individual user’s needs (e.g., 
personal annotations of learning objects or automatically adapted content). 

5.3 Conclusions of literature analysis and problems to solve 

Both analysed VLE technological evaluation tools have a number of limitations: 

■ (Technical... 2004) tool practically does not examine adaptation capabilities criteria. 

■ Graf and List (2005) tool insufficiently examines general technological criteria. 

In the authors’ opinion, the more comprehensive VLE technological evaluation tool is needed. It 
should include general ‘internal quality’ technological evaluation criteria based on the modular 
approach and interoperability, as well as ‘quality in use’ adaptation capabilities criteria. 


5.4 Recommended quality evaluation tool of virtual learning environments 

Therefore the authors have proposed an original set of VLE technological quality evaluation criteria 
combining general ‘internal quality’ and adaptation ‘quality in use’ evaluation criteria (see Fig. 2). 


General 

criteria 



f 


Overall architecture 
and implementation 


r 


Scalability 

Modularity (of the architecture) 

Possibility of multiple installations on a single platform 


Reasonable performance optimisations 
Look and feel is configurable 
Security 

M odular authentication 
Robustness and Stability 


Interoperability 


^ Installation, dependencies and portability 
Integration is straightforward 


\ VLE standards support 

Localisable user interface 
Lo c ali sation to relevant languages 
Internationalisation and localisation ) Unicode text editing and storag e 

Time zones and date localisation 


Accessibility 


c 


Alternative language support 
Text-only navigation support 


Scalable fonts and graphics 


Adaptation 

criteria 


r 


Adaptability (facilities to customize for the educational institution's needs) 


c 


Language 


Design 


Personalization aspects 

(facilities of each individual user to customize his/her own view of the platform) 




Extensibility 


c 


Good programming style 
Availability of a documented API 


Adaptivity (all kinds of automatic adaptation 
to the individual user's needs) 


r 


Personal annotations of LOs 
Automatically adapted content 


Figure 2: Technological criteria for evaluation of VLEs (Kurilovas and Dagiene 2008) 
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It is worth mentioning here that this tool includes the ‘purely technological’ VLEs evaluation criteria 
only. We can also include here a number of other ‘quality in use’ criteria which are closely 
interconnected with the ‘purely technological’ ones and suitable especially for the open source VLEs. 
These criteria are: Strength of development and users communities; Ease of installation; System 
administration; Documentation quality. 

5.5 Results of experimental technological evaluation of virtual learning environments 

The authors propose to use the multiple criteria evaluation method (usually called ‘scalarization’) 
suitable for the expert evaluation of the learning software packages such as VLEs. 

As an example the authors have evaluated the three popular open source VLEs against technological 
quality (both general and adaptation) criteria according to the VLEs evaluation tool presented in 
Figure 2 using the quality criteria evaluation ratings / values presented in Table 1. 

The authors have used the simple expert’s (decision maker’s) additive utility function 


i=\ 


( 1 ) 


where fi(a) = {0, 1, 4} is the evaluation criteria rating / value (see Table 1) for each of the three 

examined alternatives (i.e., VLEs) aj= {ATutor, llias, Moodle}, and each of the eight quality evaluation 
criteria i = {1, ... , 8}. Here i = {1, 4} are the general ‘internal quality’ VLE quality criteria, and / = 

{5, , 8} are the VLE adaptation ‘quality in use’ criteria (see Figure 2). 

The major is the meaning of the utility function (1 ) the better is the quality of the VLE. 

The results of the practical evaluation of three open source VLEs according to the expert’s additive 
utility function (1) are presented in Table 2. 


Table 2: VLE technological evaluation summary (Kurilovas and Dagiene 2008) 


Technical evaluation criteria 

AT utor 

llias 

Moodle 

General criteria 

Architecture and implementation 

Rating 2 

Rating 1 

Rating 4 

Interoperability 

Rating 3 

Rating 3 

Rating 2 

Internationalisation and localisation 

Rating 1 

Rating 2 

Rating 3 

Accessibility 

Rating 4 

Rating 1 

Rating 2 

Interim evaluation ratinq: 

10 

7 

11 

Adaptation criteria 

Adaptability 

Rating 1 

Rating 2 

Rating 3 

Personalisation 

Rating 3 

Rating 3 

Rating 2 

Extensibility 

Rating 3 

Rating 4 

Rating 4 

Adaptivity 

Rating 1 

Rating 0 

Rating 1 

Interim evaluation ratinq: 

8 

9 

10 

Total evaluation ratinq: 

18 

16 

21 


According to these practical evaluation results, Moodle is the best VLE from the technological point of 
view in general case (i.e., when all the weights of evaluation criteria values are equal). 


6. Optimisation of quaiity parameters of e-Learning software packages 

The authors have proposed 34 quality evaluation criteria suitable for the LORs multiple criteria expert 
evaluation in Figure 1. The authors have also proposed to use the quality evaluation ratings / values 
presented in Table 1. 

In this case we can use the expert’s (decision maker’s) additive utility function (1) where fi(a) = {0, 1, 
... , 4} \s the evaluation rating (value) for each of the examined alternatives (LORs) aj = {LORI, LOR2, 
..., LORn} and each of the 34 LORs quality evaluation criteria i = {1, ... , 34}. The major is the 
meaning of the utility function (1) the better is the quality of the LOR. 

In this simple case we invoke the generic principle that all the weights of evaluation ratings are equal. 
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But overall, if we want to optimise LORs / VLEs for the particular learner needs, i.e., to personalise 
his/her learning process in the best way according to their prerequisites, preferred learning speed and 
methods, etc., we can use the values of LORs and VLEs quality evaluation criteria presented in 
Figures 1 and 2 together with the experts’ preferred weights of the evaluation criteria. 

In this case we have the multiple criteria optimisation task using the criteria values and their weights. 
There are several suitable methods exist to solve such kind of tasks. For example, Zavadskas and 
Turskis (2008) have proposed logarithmic normalisation method in games theory. Dzemyda and 
Saltenis (1994) have proposed multiple criteria decision support system. Three interactive methods of 
increasing complexity were realized in this system: paired comparisons of alternatives, Pareto, and 
Fuzzy. 

The authors have further analysed the VLEs / LORs multiple criteria optimisation task using the VLEs 
/ LORs quality criteria ratings and their weights according to the particular users needs. In the authors’ 
opinion, the multiple criteria evaluation task can be successively solved using more complex experts’ 
additive utility (scalarization) function: 

m g 

5(a) = ^w,.y;.(a), = 1, w, >0, (2) 

1=1 h=l 

where W/, are the normalised weights of the evaluation criteria. 

The proposed LORs / VLEs multiple criteria evaluation method represented by the experts’ additive 
utility function (2) is based one the transformation of the multiple criteria task into the one-criterion 
task obtained by adding all criteria ratings (values) together with their weights. The authors have 
analysed the case of the choice of VLE mostly suitable for users with special needs or disabilities (see 
Table 3). In this case the authors have chosen the higher weights 0.2 for Accessibility and 
Personalisation evaluation criteria: W 4 - 0.2 and We- 0.2 (see Fig. 2). The other weights should be w,= 

g 

0.1, so that = 1. 

h=l 


Table 3: Technological evaluation summary of VLEs suitability for special needs students 


Technical evaluation criteria and weights 

AT utor 

Mias 

Moodle 

General criteria 

Architecture and implementation {wi= 0.1) 

0.2 

0.1 

0.4 

Interoperability (W 2 = 0.1) 

0.3 

0.3 

0.2 

Internationalisation and localisation (W 3 = 0.1) 

0.1 

0.2 

0.3 

Accessibility {W 4 = 0.2) 

0.8 

0.2 

0.4 

Interim evaluation rating: 

1.4 

0.8 

1.3 

Adaptation criteria 

Adaptability (ws = 0.1) 

0.1 

0.2 

0.3 

Personalisation (we= 0.2) 

0.6 

0.6 

0.4 

Extensibility {W 7 = 0.1) 

0.3 

0.4 

0.4 

Adaptivity {wa= 0.1) 

0.1 

0.0 

0.1 

Interim evaluation rating: 

1.1 

1.2 

1.2 

Total evaluation rating: 

2.5 

2.0 

2.5 


These results mean that both ATutor and Moodle are the best VLEs alternatives for special needs 
students. 


If we want, e.g., to select the most suitable LOR for the users with special education 
needs/disabilities, we should choose the higher weights for the particular criteria such as ‘Flexibility 
and modularity of the LOR system’ (wi), ‘Customisable metadata schema’ (wu), ‘Customisable and 
extensible standard Ul’ (W 24 ), ‘Accessibility’ (W 31 ), and ‘Ability to customise look and feel’ (W33) (see 

g 

Fig. 1 ). All other weights should be equal, so that = ^. 

h=l 

In this case it could be estimated that Fedora is the optimal LOR for the users with special needs in 
comparison with, e.g., DSpace and EPrint due to its modular approach, metadata schema which is 
extensible without restrictions, its high ability to customise look and feel, etc. 


www.ejel org 


149 


ISSN 1479-4403 




Electronic Journal of e-Learning Volume 8 Issue 2 2010 (141 - 150) 


7. Conclusion 

The comprehensive LORs and VLEs quality evaluation tools should include both general software 
‘internal quality’ evaluation criteria suitable for all software packages quality evaluation cases, and 
‘quality in use’ evaluation criteria suitable for the particular project or user. These tools are based on 
sound scientific principles, they are more comprehensive and therefore they are more suitable in 
comparison with the other learning software quality evaluation tools. The authors propose using the 
experts’ additive utility (scalarization) function (2) to solve the LORs and VLEs multiple criteria 
optimisation tasks for the individual learner needs. This method is of practical importance for 
educational sector’s experts, software engineers, programmers and users. 

8. Appendix 

The work presented in this paper is partially supported by the European Commission under the 
eContentp/us programme - as part of the EdReNe project (project number ECP-2006-EDU-42002). 
The authors are solely responsible for the content of this paper. It does not represent the opinion of 
the European Commission, and the European Commission is not responsible for any use that might 
be made of data appearing therein. 
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